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Executive Summary 



In 2002, the Tennessee Department of Education asked the Appalachia Educational 
Laboratory (AEL) at Edvantia, Inc. to develop content for online professional development that 
would help meet the goals of the state’s Reading First program. In response, laboratory staff 
developed and subsequently delivered Assessment and Intervention in a Comprehensive Literacy 
Classroom, a five-module course delivered over 16 weeks. The course was designed to help K-3 
teachers, K-12 special education teachers, and building-level administrators incorporate formal 
and informal assessments into their reading instruction and use the results of those assessments 
to guide instruction. The course was organized and delivered in five modules that covered the 
following topics: (1) a conceptual model of a comprehensive literacy classroom in which all 
children can leam to read by the end of the third grade; (2) the five essential elements of reading 
instruction; (3) the Tennessee assessment system and the four types of assessments (screening, 
diagnosis, outcome, and progress monitoring); (4) assessment instruments, implementation steps, 
and practice opportunities; and (5) use of assessment data to guide planning, grouping for 
instruction, and selection of instructional strategies (Ross, Thigpin, Cavalluzzo, Guzman, & 
Patterson, 2004). 

The course was initially delivered in the spring of 2004 to staff at 56 elementary schools 
in Tennessee that received Reading First grants. Researchers at The CNA Corporation (CNAC) 
collaborated with the course developers and implementers to conduct a formative assessment of 
the online course. The first assessment report (Ross, et al., 2004) examined qualitative data from 
four sources for each of the participating schools: (1) online evaluations by course participants, 
(2) threaded discussions of course participants, (3) focus groups of literacy leaders, and (4) 
discussions with literacy leaders. These data revealed high levels of satisfaction with the content 
of the training modules. In addition, statistical comparisons of teacher test scores before the 
program began and after the completion of each module found substantial improvements in 
teacher knowledge in each area. 

Formative evaluation also identified two problems with the delivery of the course: (1) 
introduction of the program in the middle of the school year and (2) technology and software 
difficulties experienced at the school level. As a result of this evaluation, the Tennessee 
Department of Education shifted the program introduction to the beginning of the school year for 
subsequent administrations of the course. The technology issues were often school-based and 
related to outdated hardware and software and limited bandwidth. These issues were most often 
addressed through communications between Edvantia technical support staff and district or 
school technology support staff. 

This paper extends the prior analysis in two ways. First, it revisits the pre- and posttest 
data to estimate the effect size associated with teacher participation in the online program and 
examines changes in student outcomes in the 56 Reading First schools relative to a sample of 
matched comparison schools. Second, the paper offers a cost analysis of development and 
delivery of the online program and compares it to the projected cost of delivery of the same 
materials through face-to-face workshops. The paper considers these costs from the perspectives 
of (1) the developer; (2) the state, which paid for course development; and (3) the schools, which 
purchased course seats for their teachers. 
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Findings 



Impact on teacher knowledge. Researchers found strong evidence that the online 
professional development course had an immediate and important impact on teacher knowledge 
pertaining to the use of assessments to guide literacy instruction. Comparisons of test scores 
prior to and immediately following completion of the training modules showed that overall 
scores increased 21 percentage points on average. The effect size associated with this 
improvement is 1.6 standard deviation units, a magnitude that is widely recognized as large and 
important in the field of education (Cohen, 1988). 

Impact on student achievement. Due to limitations in the public availability of 2005 
state student achievement data, researchers were not able to fully explore the relationship 
between teacher gains from participation in the online course and their students’ academic 
achievement. However, researchers did compare schools that received the professional 
development training to a matched sample of control schools in which teachers did not receive 
training. Specifically, researchers assessed whether the Reading First intervention schools, 
compared to non-Reading First control schools, differed in the proportion that met the state and 
federally mandated 2003, 2004, and 2005 adequate yearly progress (AYP) goals. While both the 
intervention and control schools dramatically improved their ability to achieve AYP between 
2003 and 2005, the improvement was larger for intervention schools. This suggests a positive 
impact of the reforms in the Reading First schools. Due to the timing of the delivery of the online 
course and in-school improvement, however, the relationship between the intervention and the 
observed improvement is unclear. Once 2005 state test scores for reading become available by 
grade level and school, it will be possible to revisit the question of intervention impact on student 
achievement. 

Cost effectiveness. The analysis of cost data shows that providing the online course is an 
inexpensive way to furnish professional development and compares favorably to projected costs 
associated with face-to-face delivery of the same materials. Moreover, a key design feature of 
the online program is that it spreads instruction over a 16- week period. The least expensive face- 
to-face instruction, in contrast, would require a five-day training session to cover the same 
materials. However, delivery of professional development in short bursts, rather than over time, 
has been shown to have little or no effect on teaching practices (Supovitz, 2002). 

Alternatively, face-to-face instruction could be delivered over time using a train-the- 
trainer approach. Such an approach would be most similar to the online design but would be 
substantially more expensive to implement if trainers had to travel long distances, as they would 
in Tennessee, to receive training in five intervals. Moreover, the evidence assembled here 
suggests that the online program resulted in large improvements in teacher knowledge and was 
substantially more costeffective when compared to face-to-face instruction. 
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Introduction 



Online professional development is a subset of distance learning in general and, as such, 
has seen tremendous growth in terms of technologies, pedagogy, and research over the past 10 
years. In this past decade, numerous colleges and universities have undertaken efforts to make 
their entire graduate and undergraduate course catalogues available online, and new virtual 
universities, such as the University of Phoenix and Western Governors University, have become 
major degree-granting institutions. Virtual high schools have been established — either for profit 
or as extensions of public school offerings — in more than a dozen states. Online professional 
development has become a substantial growth market. Despite this, there is little indication in the 
experimental research literature of the relationship between cost and effectiveness of online 
professional development. Some measures used in evaluating distance learning in general 
include student outcomes as well as faculty and student perceptions (Ramage, 2002). 

The concept of online learning, or e-Learning, as it is sometimes called, has received 
support from the U.S. Department of Education for its ability to help schools achieve several 
goals of the No Child Left Behind Act. These include providing increased access to high-quality 
professional development for teachers, paraprofessionals, and principals as required specifically 
by Title II of the Act and by many subsections throughout the law (e.g., Reading First, 

Enhancing Education through Technology). The July 12, 2002, press release from the U.S. 
Department of Education, Paige Touts Technology, E-Learning on His No Child Left Behind 
Tour Across America (U.S. Department of Education, 2002) and the review of the July 21, 2003, 
forum from the American Youth Policy Forum, Online teaching, online learning: A cost effective 
choice for states? are but two accounts from John Bailey, former director of educational 
technology at the U.S. Department of Education, and former Education Secretary Rod Paige that 
support e-Learning. Additional support for online professional development by Department 
personnel can also be found (Bailey, 2002; Bailey, 2003; Plato, 2003). 

The reauthorization of the Elementary and Secondary Education Act, otherwise known as 
the No Child Left Behind Act (NCLB) of 2001, creates opportunities and challenges for schools. 
One opportunity is increased funding for state-designed literacy grants through Reading First. 
With this funding, states can design and offer reading programs to improve the skills of K-3 
principals and teachers and K-12 teachers of special education. Concomitant with this 
opportunity is the challenge of the Reading Excellence Act (REA) of 1999 that all schools have 
highly qualified teachers by 2006. To meet this challenge, teachers can turn to college 
coursework or professional development. States with a significant number of rural school 
districts, such as Tennessee, are seeking nontraditional solutions to providing high-quality, 
research-based professional development so teachers do not need to leave their homes and 
classrooms for extended periods of time to travel to distant institutions to hone and develop 
skills. Tennessee chose to provide professional development via distance learning and turned to 
the Appalachia Educational Laboratory (AEL), at Edvantia, to develop and deliver a course. The 
course Assessment and Intervention in a Comprehensive Literacy Classroom was the result of 
this collaboration. 

This paper examines evidence of the effectiveness and costs of the course, based on 
implementation in 56 schools in the spring of school year 2003-2004. The analysis focuses on 




three areas: (1) the impact of online professional development on teacher learning, (b) the impact 
of online professional development on student achievement, and (3) the overall cost 
effectiveness of online professional development. The goal of this analysis is to help educators, 
policymakers, and other stakeholders make informed decisions about online professional 
development and determine under what circumstances it is most effective. 

Characteristics of Effective Professional Development Models 

Professional development has been broadly defined by the U.S. Department of Education 
(1996) as “the rigorous and relevant content, strategies, and organizational supports that ensure 
the preparation and career-long development of teachers and others whose competence, 
expectations, and actions influence the teaching and learning environment” (p. 1). This statement 
conveys the important role of professional development in the advancement of learning. Its 
importance is further underscored by the amount of literature that exists on the topic. Over the 
past two or three decades, several attempts have been made to synthesize this material, but the 
task continues to become more difficult as the number of methods and professional development 
models proliferate (Sparks & Loucks-Horsley, 1989; Sprinthall, Reiman, & Theis-Sprinthall, 
1996). 



While many studies have looked at what constitutes effective professional development 
for teachers, very little of the research can be verified statistically through empirical data (Wilson 
& Berne, 1999). Part of the explanation for the scant number of empirical studies can be 
explained by the difficult and complicated nature of social sciences research. However, the 2001 
No Child Left Behind reform blueprint (NCLB, 2002) has established high standards for the 
professional development of America’s teachers by requiring states and school districts “to 
ensure that federal funds promote the use of scientific, research-based and effective practice in 
the classroom” (p. 13). 

In 2005, the Appalachia Educational Laboratory (AEL) conducted a review of existing 
empirical studies of both face-to-face and online professional development programs for the 
period of 1985 to 2005 (Larson, in press). More than 100 abstracts, articles, reports, and 
dissertations were analyzed to find studies with conclusions based on research designs that were 
either experimental or quasi-experimental with a matched control group and clearly defined data 
measures. Twenty-one of these studies met the criteria for inclusion in this analysis of the 
components that make up effective professional development programs. A full list of these 21 
research projects is provided in Appendix A. Evidence was gathered to support the identification 
of nine components of effective professional development as extracted from the 21 studies. Some 
components were supported by more evidence than others, as illustrated in Table 1. The nine 
components identified indicate that effective professional development is 

1 . linked to student learning outcomes 

2. job embedded 

3. ongoing and sustained, with follow-up 

4. incorporates authentic, active learning experiences 

5. includes subject-matter content 

6. encourages reflection on pedagogy, content, and beliefs 
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7. incorporates collaboration with colleagues and/or experts 

8. provides support for teachers 

9. measures impact on student achievement 



Table 1. 

Effective Professional Development Components and Corresponding Numbers Supporting Studies 



Professional Development Components 


Number of 
Supporting 
Experimental 
Studies 


Number of Supporting 
Quasi-Experimental 
Studies 


Total Number 
of Supporting 
Studies 


Linked to student learning outcomes 


5 


13 


18 


Job embedded 


5 


13 


18 


Ongoing and sustained, with follow-up 


5 


8 


13 


Incorporates authentic, active learning 
experiences 


6 


11 


17 


Includes subject-matter content 


7 


13 


20 


Encourages reflection on pedagogy, 
content, and beliefs 


7 


14 


21 


Incorporates collaboration with colleagues 
and/or experts 


5 


12 


17 


Provides support for teachers 


6 


10 


16 


Measures impact on student achievement 


5 


11 


16 



Evidence of the Effectiveness of Distance and Online Instruction Versus Face-To-Face 
Instruction 

Educators, policymakers, and other decision makers have raised concerns about what 
comprises online professional development, the infrastructure it requires, and its costs and 
benefits — especially when compared to face-to-face delivery (Richardson, 2001; Watkins, 2003; 
Yoder, 2001). Common concerns relate to whether online professional development is as 
effective as face-to-face delivery or whether participants in online professional development may 
feel isolated. Benefits are also mentioned, such as greater availability of topics, convenience and 
flexibility in scheduling, and more opportunity for follow-up — often through participation in 
online learning communities. 

More than 2,000 studies can be identified that deal with distance education. However, 
most focus on courses for academic credit — graduate, undergraduate, and even high school 
courses. Distance-based professional development opportunities are generally absent from the 
research literature (Bernard et al., 2004, p. 390). Nonetheless, several studies suggest that there 
is no significant difference between the outcomes of online and face-to-face instruction. Ryan 
(2002), in the study of one undergraduate mathematics course, found no significant difference, as 
measured by final grade, between classroom, telecourse, and Web-based delivery systems. 
Expanding the scope of the study to the college’s entire catalog of distance education offerings, 
the investigator again found no significant statistical difference in either the attrition rates or the 
grade point averages of the participants in online courses compared to those in face-to-face 
settings. Karr, Week, Sunal, & Cook (2003) compared student performance in a graduate 
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engineering mathematics course that was offered online, traditionally, and as a hybrid online/in- 
class experiment. Assessing student performance on three tests, one final examination, and 
homework assignments, the researchers concluded that mode of delivery had little impact on 
student outcomes. 

A meta-analysis by Bernard et al. (2004) provides a comprehensive survey of the 
empirical literature that compares distance learning and classroom instruction. 1 In all, 
achievement, attitude, and retention outcomes from 232 studies were analyzed. The researchers 
found that the literature often lacked depth, stating: “Nearly 60 percent of codable study 
features, including methodological features, were coded as missing” (p. 407). The investigators 
also rated the methodological quality of the literature as “weak,” noting that field experiments 
are “often conducted in circumstances in which opportunities to control for rival explanations of 
research hypotheses are minimal” (p. 407). The survey suggests that “many applications of DE 
[distance education] outperform their classroom counterparts and that many perform more 
poorly” (p. 379). The Bernard et al. (2004) meta-analysis generally supports the claim that 
effective distance education depends on the provision of pedagogical excellence. 

AEL’s Tennessee Online Professional Development Course 

Developers report that the Tennessee professional development course in reading was 
designed to provide an online experience that capitalized on the best research available. Special 
attention was paid to developing learning communities, addressing the needs of adult learners, 
and to emphasizing reading-specific content and skill acquisition over technical skills. 
Guidelines for content development included internally developed instructional design 
guidelines, as well as application of Keller’s (1987, 1999) ARCS model of motivational design 
(presented later). 

Based on Edvantia’s 40 years of experience in the effective delivery of professional 
development and the work of organizations like the National Staff Development Council (2001), 
an internal work group developed instructional design guidelines for professional development 
that influenced the development of the online content (Table 2). These standards emphasize 
effective strategies such as the development of learning communities, providing sustained 
support over time, and acknowledging the needs of adult learners, regardless of delivery mode. 



The primary methods of delivery in distance education have changed at least five times over 20 years (Taylor, 
2001). Bernard et al.’s analysis encompasses these technologies. 
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Table 2. 

Instructional Design Guidelines 



Guideline 


Description 


Lramework 


Learners are better able to make connections and derive personal 
meaning from content when the instructional design includes a 
framework or structure for that content. 


Evidence 


Adult learners look for an evidence base when they approach a new 
learning experience. 


Relevance 


Relevant content and opportunities for practical applications enhance 
adult motivation and learning. 


Community 


Learning occurs best in community. 


Time 


Learning occurs over time. 


Learner background 


Effective learning designs acknowledge and accommodate the diverse 


and preference 


backgrounds, interests, and learning styles of adults. 


Activities 


An appropriate balance of content structuring (presentation); personal 
reflection; interaction with colleagues; and hands-on, or applied, learning 
enhances adult motivation and engagement. 


Assessments 


Lormative assessment is integral to participant learning and improving 
the instruction. 



Because of a history of low completion rates for self-paced instruction, the Tennessee 
Department of Education and the course developers decided not to employ completely self-paced 
instruction during the implementation. Building on the learning community metaphor, the state 
designated master teachers at each school (called “Literacy Leaders”), who received training 
during the year on a variety of topics related to reading instruction and assessment. These 
Literacy Leaders served as facilitators for their own learning communities, thereby creating what 
might be referred to as “group self-paced instruction,” as each community enforced its own 
timeline, level of participation, and acceptable levels of content mastery. The content included 
structures intended to guide the Literacy Leaders, as well as face-to-face and online orientation 
programs, which presented guidelines for obtaining requisite administrative and technical 
support, developing and keeping a timeline, and facilitating online learning. 

ARCS Model. Developed by Keller (1987), the Attention, Relevance, Confidence, and 
Satisfaction (ARCS) model of motivational design is widely used in the development of 
instruction in multiple formats (Small, 1997), including computer-based instruction and distance 
education (Keller, 1999). Each of the ARCS components is subdivided into three 
subcomponents (for an overview of components, see Small, 1997). 

To gain student attention, provide relevant content and activities, build confidence, and 
offer opportunities for success the ARCS model was reflected in several facets of the 
instructional design of the course. These include the structure, chunking, and sequencing of the 
content; the design and implementation of the activities; and the selection of media and alternate 
media formats that sought to satisfy a variety of platforms, processing capacities, and connection 
speeds. The participants’ experience and current practice were critical factors as the school- 
based communities reviewed findings from research, explored their own practices, and 
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constructed new knowledge within the context of their professional learning communities. The 
activities in each module often followed a three-part progression, beginning with (1) reflection 
on initial practice, followed by (2) exploration of new concepts, and concluding with (3) 
synthesis of new skills and knowledge with current practice. These activities resulted in 
strategies or processes to apply within the school or classroom. The content of the online course 
included text, graphics, videos, and animations delivered via the course management system 
software. 

The next section of this report turns to an analysis of the effectiveness of the online 
course. Because the course has not been implemented in a face-to-face mode, the researchers 
assume, based on the review of the literature, that the face-to-face course would be equally 
effective if implemented in a comparable way — that is, as an ongoing process that engages a 
group of teachers in the learning experience. 

Indicators of Course Effectiveness 

To draw conclusions about the effectiveness of the online program, this section presents a 
review of the results from analyses of data related to teacher learning. A comparison of online 
and face-to-face delivery within the framework of the nine effective components identified 
through a review of the professional development research literature is also provided. 

Teacher Learning 

Researchers used pre- and posttests developed for the course to examine the impact of 
each module on participants’ understanding of the materials. The tests developed for the course 
consist of one 68-question pretest and a matched series of five posttests totaling 68 questions. 
Questions on the pretest and each posttest correspond to the learning objectives for the five 
modules. A total of 1,270 classroom teachers and administrators took the online module tests. Of 
these, 693 participants completed all pretest and corresponding posttest questions. Based on this 
sample, the average pretest score was 64%. Average pretest scores across modules ranged from a 
low of 51% in Module 4 (assessment instruments) to a high of 75% in module 5 (use of 
assessment data to guide planning, grouping for instruction, and practice opportunities). Taken 
together, scores improved an average of 21 percentage points across all five Modules, with the 
greatest gains in Modules 4 (31 percentage points) and 3, which dealt with the assessment 
process and types of assessments (27 percentage points). Posttest average scores, shown in Table 
3, were greater than 80% for every module. Researchers conducted paired /-tests of pre- and 
posttest scores to confirm the statistical significance of these results. For each module and for the 
68 questions overall, the gains were significant (< .01). In addition, using Cohen’s d, researchers 
assessed the magnitude of the pre-post differences for each module and across the full set of 68 
items. Analyses on the overall pre-post scores revealed an effect size of 1.6 standard deviation 
units. For specific modules, effect sizes ranged from a low of .75 for Module 5 to a high of 1.43 
for Module 3. Effect sizes of .8 standard deviations or more are widely regarded as large in the 
field of education (Cohen, 1988). Using this guideline, the intervention appears to have been 
quite successful. Table 3 presents average pretest and posttest scores and percentage gains by 
module. 
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Table 3. 

Participants’ Pre- and Posttest Scores by Module (N = 693) 



Module 1 


Module 2 


Module 3 


Module 4 


Module 5 


Total % Correct 


Correct pre 


63% 


64% 


61% 


51% 


75% 


64% 


Correct post 


83% 


83% 


89% 


82% 


88% 


85% 


Percentage point gain 


20 


19 


27 


31 


13 


21 


t test 


27.48 


27.59 


37.58 


32.85 


19.78 


42.49 


Effect size 


1.04 


1.05 


1.43 


1.25 


.75 


1.62 


N of questions 


12 


20 


13 


9 


14 


68 



Impact of Training on Student Achievement 

To examine the impact of the training on student achievement, the initial plan was to 
compare schools receiving training to a set of matched comparison schools on their third-grade 
Dynamic Indicators of Basic Early Literacy Skills (DIB ELS), Tennessee Comprehensive 
Assessment Program (TCAP), and Tennessee Value-Added Assessment System (TVAAS) 
scores. DIBELS data were not made available for the study. At the time of this report, the 2005 
TCAP and TVAAS data are not publicly available. Instead of these data, the researchers used the 
schools’ Adequate Yearly Progress (AYP) ratings in 2003, 2004, and 2005 as crude indicators of 
improvement in student performance that may have been influenced to some degree by teacher 
participation in the online professional development course in reading. 

Preceding from the assumption that professional development impacts student 
achievement through improvements in teacher knowledge, which enhances classroom practices 
and, subsequently, student outcomes, the researchers considered the potential for impact of the 
intervention on the AYP status of each school. Given the importance placed on attaining AYP 
under No Child Left Behind, this outcome is important to examine in the overall context of the 
effectiveness of any Reading First program. However, a wide variety of factors affect a school’s 
ability to achieve AYP, and multiple reforms may have been introduced to improve outcomes in 
the Reading First schools. Furthermore, AYP data are presented at the overall school level and 
not by individual grade in school. Therefore, these analyses are only suggestive of the impact 
that the course may have had on student literacy. 

For the following analyses, researchers matched each of the 56 schools that participated 
in the reading course with a comparison school from within the same county, and then compared 
schools on three variables: (1) percentage of White students (2) percentage of economically 
disadvantaged students, and (3) overall school size. The data for selecting the comparison 
schools were collected from the SchoolMatters Web site (http://www.schoolmatters.org). For 
one of the 56 schools, general demographic and student performance information were not 
publicly available. Therefore, the school was omitted from subsequent analyses. Researchers 
conducted paired sample t tests on the 55 matched schools to confirm that the intervention 
schools were similar to their matched comparison schools on the dimensions specified. As shown 
in Table 4, the analyses revealed no significant differences, on average, between the intervention 
schools and their matched counterparts on the three selection variables; on average, the 
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percentages of White students and economically disadvantaged students were in each type of 
school within five percentage points of each other. 

Outcome data reported in this analysis are from the Tennessee Department of Education 
2003 and 2004 Report Cards. Between 2003 and 2004, the state instituted a new test of student 
performance and redefined the benchmark by which AYP is assessed. Although a potential 
confound for this analysis, these changes are assumed to influence the intervention and control 
schools to the same degree. Therefore, any observed differences between intervention and 
control schools are suggestive of a possible impact of the Reading First intervention. 



Table 4 

Means and Standard Errors of Selection Variables for Schools Receiving Training and Matched 
Comparison Schools 





Schools Receiving 
Training 


Matched Comparison paired samples t test 
Schools (df = 54) 


P 


Percentage of White Students 


32.0% 


33.7% 


-0.79 


>.05 


Standard Error 


5.0% 


4.7% 






Percentage of Economically 
Disadvantaged Students 


85.3% 


83.8% 


0.80 


>.05 


Standard Error 


1.7% 


1.8% 






School Size 


403.5 


448.9 


-1.93 


>.05 


Standard Error 


26.6 


26.5 







Analyses were conducted in two parts. First, researchers compared the proportion of 
intervention schools making AYP to the proportion of comparison schools making AYP for each 
of the three years (2003, 2004, 2005). Second, they conducted separate analyses for intervention 
and comparison schools to assess any longitudinal change in the proportion of schools making 
AYP. For all reported analyses, the McNemar chi-square test was used. The McNemar test is a 
common nonparametric test for two related dichotomous variables (Armitage & Berry, 1994; 
Liddell, 1983) and appropriate for the analysis of both matched pairs and repeated-measures 
dichotomous data. This test is particularly useful for detecting changes in responses due to 
experimental intervention in before-and-after designs. Finally, all reported test statistics are from 
the marginal homogeneity test, which is an extension of the McNemar test. This statistic tests 
for changes in response using the chi-square distribution and is useful for detecting response 
changes due to experimental intervention in before-and-after designs. To facilitate understanding 
and interpretation, the data are presented in terms of the percentage of intervention or 
comparison schools that achieved AYP in a given year. Figure 1 shows the percentage of 
intervention and comparison schools making AYP in the preintervention year (2003), 
intervention year (2004), and postintervention year (2005). 
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Comparison of Intervention to Control Schools 

Analyses revealed that in the pre-intervention year (2003), intervention schools had a 
significantly lower probability of making AYP (Achieved AYP = 12%) compared to comparison 
schools (Achieved AYP = 36%), % 2 (i) = 8.33; p < .006. In the intervention year (2004), analyses 
revealed no significant differences between the intervention schools (Achieved AYP = 76%) and 
comparison schools (Achieved AYP = 74%), %'(i) = 0.09; p > .05. Finally, in the 
postcomparison year (2005), analyses also revealed no significant differences between the 
intervention schools (Achieved AYP = 82%) and comparison schools (Achieved AYP = 93%), 
% 2 (i) = 2.00; p > .05. 




Figure 1 

Percentages of Intervention and Comparison Schools Making AYP in 2003, 2004, and 2005 



Longitudinal Changes in Making AYP 

Intervention schools. Analyses revealed a statistically significant increase in the 
proportion of intervention schools that achieved AYP in 2004 Achieved AYP = 76%) compared 
to those that achieved AYP in 2003 (Achieved AYP = 12%), x 2 (i) = 31.00; p < .0001. Analyses 
also revealed no statistically significant differences between the postintervention year, 2005 
(Achieved AYP = 82%) and the intervention year, 2004 (Achieved AYP = 76%), % 2 (i) = 2.27; p 
> .05. 

Comparison schools. Analyses revealed no statistically significant differences between 
the proportion of comparison schools that achieved AYP in 2004 (Achieved AYP = 74%) and to 
those that achieved AYP in 2003 (Achieved AYP = 36%), x 2 <n = 1.00; p > .05). Analyses also 
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revealed no significant differences between the postintervention year, 2005 (Achieved AYP = 
93%), and the intervention year, 2004 (Achieved AYP = 74%), % 2 (i) = 0.14; p > .05. 

In summary, the matched comparison analyses revealed significant differences between 
the intervention and comparison schools. First, direct comparisons between the intervention and 
comparison schools revealed that intervention schools had a significantly lower probability of 
achieving AYP in 2003. However, no significant differences were observed between 
intervention and comparison schools in either the intervention (2004) or postintervention (2005) 
years. That is, the probability of intervention schools achieving AYP did not significantly differ 
from that of the comparison schools following the online training. Second, significant 
differences between intervention and comparison schools were also observed in the longitudinal 
analyses. For the intervention schools, the probability of achieving AYP significantly increased 
from the preintervention (2003) to the intervention year (2004). No significant differences were 
observed from the intervention (2004) to the postintervention year (2005). For the comparison 
schools, no significant differences were observed for either the preintervention (2003) to 
intervention year (2004) comparison, or the intervention (2004) to postintervention year (2005) 
comparison. While these observed differences are likely due in part to the new Tennessee state 
assessments introduced in 2004, it is also important to note that these results are suggestive of 
the impact of reforms in the Reading First intervention schools, including the possible positive 
effects of the online professional development course. 

Comparison of Delivery Methods Across Characteristics of Effective Professional 
Development Models 

The course developers identified nine characteristics of effective professional 
development models through a review of research studies, the conclusions of which were based 
on research designs that were either experimental or quasi-experimental, with a matched control 
group and clearly defined data measures. Although the spring 2004 administration of the reading 
professional development was offered only online, the same content, activities, and materials 
could be used to deliver the training in a face-to-face setting. With more than 40 years of 
experience designing and delivering professional development in a face-to-face setting, Edvantia 
developers are fully cognizant of the requirements and projected costs for doing so. 

For the purposes of this comparison, the developer estimated that five days would be 
required to deliver a comparable face-to-face course. Facilitators would include one research and 
development specialist and one senior research and development specialist. An event of this 
duration would, most likely, occur when classes are out of session; otherwise, additional costs 
related to substitutes would be involved. An event of this duration while classes are in session 
would also limit the number of teachers who could attend, as few schools would release all of 
their K-3 teachers for the same five days. It is possible that the initial staff development could 
occur over a span of three days with an additional follow-up of one two-day event or two one- 
day events; however, the costs of supporting facilitator travel and lodging remain fairly equal for 
each trip, so an initial event with follow-up was not considered to be a cost-effective option. 

Table 5 indicates the number of characteristics associated with the two delivery modes 
for the current professional development. As can be noted, the online delivery method exhibits 
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more components of effective professional development models as identified by research 
(Larson, in press) when compared to the potential face-to-face delivery of the same course; 
online delivery exhibits eight of the nine components and face-to-face delivery potentially 
exhibits 5 components. 

Table 5 

Effective Professional Development Components and Modes of Delivery 

Professional Development Components Online Delivery 

Linked to student learning outcomes * 

Job embedded * 

Ongoing and sustained with follow-up * 

Incorporates authentic, active learning experiences * 

Includes subject-matter content * 

Encourages reflection on pedagogy, content, and beliefs * 

Incorporates collaboration with colleagues and/or experts * 

Provides support for teachers * 

Measures impact on student achievement 
Components of effective professional development models 

The components that do not exist in the face-to-face delivery option — job embedded, 
ongoing and sustained with follow-up, and provides support for teachers — are well-suited to the 
online mode. In this case, teachers did not need to leave their classrooms to participate in the 
online training. They could, in fact, learn and practice new skills and knowledge outside of class 
and return the next day to incorporate them into their teachings. This aspect of the online 
approach makes it highly job embedded. The course was, by its very nature, ongoing and 
sustained with follow-up because it was spread over a period of 16 weeks rather than isolated in 
five days. In addition, participants have access to the online course materials, including all 
content and videos, for up to a year, making it possible to have “refresher” sessions at the start- 
up of the following school year. Ample pedagogical and technical support were provided to 
teachers throughout the 16 weeks, as literacy leaders participated in a separate and private online 
discussion forum to share ideas and discuss methods for effectively disseminating the content. In 
addition, reading experts from Edvantia, the Tennessee Department of Education, and the state’s 
reading cadre, which consists of faculty members from institutions of higher education, were 
also able to access the course material and communicate directly with the schools. Critical 
technical support was available 24/7 from Avatar, and free phone and online support were 
available to all participants through Edvantia tech support staff. Participants’ perceived success 
of and appreciation for the timeliness and effectiveness of this technical support was a key 
finding in the previous report (Ross, Thigpin, Cavalluzzo, Guzman, & Patterson, 2004). 

Summary 

As noted earlier, the assessed impact of the online professional development on the 
schools’ ability to achieve AYP is a crude indicator of the potential impact of the training on 
student achievement. The findings suggest a positive impact from reforms in the Reading First 
schools. Due to the timing of gains and of the online professional development program, it is 
unclear whether any causality can be attributed to those observed gains. 
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Based on a comparison of the nine characteristics of this training with those deemed 
effective through a review of rigorous research on professional development, the online delivery 
method exhibits eight of these characteristics. Face-to-face delivery exhibits only five. 

The Cost of the Online Education Program 

In this section the costs of the online professional development program are analyzed 
from the perspectives of the developer and provider, the state, and the schools. Those costs are 
then compared to the sale or purchase of the course in a face-to-face environment. 

The Full Cost of Developing and Providing the Online Curriculum to Program Participants 

In its first year, the online professional development program was completed by 893 
teachers in 56 schools. The costs to develop and deliver the course are of two main types: fixed 
costs and variable costs. Fixed costs are those expenses that do not change with the amount of 
services provided (course seats in this case). Variable costs, in contrast, increase as more course 
seats are provided. It is common in the provision of online education to have a cost structure in 
which fixed costs are substantial and variable costs are negligible. As a result, it is cost effective 
to offer courses online only when large numbers of students are expected to use the courseware. 
Researchers obtained cost data for development and delivery of the online course from Edvantia 
and, based on additional discussions to clarify their purposes, allocated expenditures to fixed and 
variable costs. Table 6 displays the fixed costs associated with provision of the online course. 

Table 6 

Fixed Cost of Development and Delivery 



Cost Component 


Estimated Fixed Costs 


Development 


$59,711 


Delivery 




Workshop 


$1,532 


Platform service fee 


$3,000 


Total fixed costs 


$62,243 


Average fixed cost per participant for 896 course seats 


$72 



Courseware development is the greatest source of fixed costs. The course provider spent 
$59,711 to develop the courseware. Costs for development include salaries for content experts, 
instructional designers, and multimedia developers who developed all graphic, video, and Web 
elements; all materials and supplies, including any software or hardware required during 
development; and indirect costs associated with facilities operation. The developer states that 
development costs would be similar, whatever delivery method is selected. 

Delivering the course also involves costs that are largely fixed in nature. To deliver the 
course, the provider contracts with a platform provider. Avatar, to make the courseware 
accessible to students. The platform provider charges an annual flat fee of $3,000 for an 
unlimited level of use, plus $10 per course seat. The platform expense, then, has both fixed and 
variable components. Another fixed expense for this one administration of the course came from 
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a three-hour workshop that was offered to all 56 literacy leaders. Because this expense was 
driven by the cost of the facility and the instructor’s time and travel expense, it would have been 
no different if fewer literacy leaders had enrolled in the course. The cost of the three-hour 
workshop was $1,532. In total, these expenses sum to $67,243, or $71.70 per course seat, when 
averaged across the 896 participants who completed the course. 3 

When considering variable costs, it has been mentioned that the platform provider 
charges $10 per course seat to provide access to the courseware via the Internet. In addition, each 
literacy leader receives training materials valued at $30 each. Rather than increasing with each 
additional course seat, this expense increases with each additional school that enrolls. Table 7 
displays variable costs under the assumption that, on average, 16 participants enroll from each 
school. 



A final variable cost comes from the personnel who support the program. These people 
handle administrative functions such as enrollments and record keeping. They also support 
course participants, primarily by answering questions and solving technical problems as they 
arise. These expenses sum to $3,340 for salaries and indirect costs and are not expected to differ 
substantially over wide ranges of enrollments. This expense is thought to be rather insensitive to 
the number of participants who enroll. We have assumed that this expense increases 30% for 
each 100% increase in enrollment. Totaled across expense categories, we find that variable costs 
for the program were $13,980 for 896 participants from 56 schools, and those costs would rise to 
$25,622 if the number of schools and participants doubled. Because the cost of administrative 
support does not rise as quickly as enrollments, the variable cost per participant drops somewhat, 
from $15.60 per course seat to $14.30 per course seat, if enrollment doubles. 

Table 7 

Variable Costs for Providing Training for Three Levels of Enrollment 



Variables 


Enrollment 
Level 1 


Enrollment 
Level 2 


Enrollment 
Level 3 


Number of schools 


56 


112 


224 


Number of course seats* 


896 


1,792 


3,584 


Variable costs 








Course seats 


$8,960 


$17,920 


$35,840 


Training materials 


$1,680 


$3,360 


$6,720 


Salaries and indirects** 


$3,340 


$4,342 


$5,645 


Total variable costs 


$13,980 


$25,622 


$48,205 


Average variable costs 


$15.60 


$14.30 


$13.45 



2 

Although a larger number of literacy leaders might have led to the offering of a second 3 -hour session, we do not deal with that 
complication because, in fact, the developer has dispensed with the literacy leader workshop. Thus, it was a one-time expense 
associated with the piloting of the course. 

3 

In fact 893 participants completed the course, or 15.95 participants per school. To simplify discussion, we round usage to 896 
in our analyses, or 16.0 participants per school. The change has no meaningful impact on the cost analysis. 



13 






* Assumes 16 participants per school on average 

* *Estimate assumes a 30% increase in support costs associated with a 100% increase in course 
enrollments 

Summing fixed and variable costs produces the full cost to the provider of developing 
and delivering the online course. These costs are shown for three different enrollment levels in 

Table 8 

Actual and Projected Cost of Online Course Development and Delivery by Enrollment 



Costs 


For 896 Participants 


For 1,792 Participant 


For 3,584 Participant 


Total fixed cost 


$64,243 


$64,243 


$64,243 


Total variable cost 


$13,980 


$25,622 


$48,205 


Total cost 


$78,223 


$89,865 


$112,448 


Cost per seat 


$87.30 


$50.15 


$31.38 



The full cost per course seat is $87.31 for 896 participants, $50.15 for 1,792 participants 
and $31.38 per seat for 3,584 participants. Those costs would drop if enrollments increase. 

Costs Incurred by the Education System 

Each participant course seat was sold for $25, i.e., $22,400 for 896 participants. In 
addition, in this pilot project, the state of Tennessee subsidized development of the courseware, 
paying $56,000 of those initial expenses for total out-of-pocket expenses of $78,400, or $87.50 
per participant. Doubling enrollments would double the payments for course seats but would 
have no effect on development costs. As a result, full costs per seat for 1,792 seats would drop to 
$56.25 per participant. 4 

Costs of Face-to-Face Delivery 

The developer estimates that an alternative, five-day, face-to-face course could be 
delivered at a projected cost of $7,009 per event. This cost assumes only fully-loaded staff 
salaries and travel expenses and is based on one research and development specialist and one 
senior research and development specialist serving as facilitators. The number of participants per 
school and the number of schools that participate in each event affect average costs. If the course 
could be taught in an auditorium that accommodes 100 participants, the projected delivery costs 
would be $70 per person. Using the average number of 16 teachers per school, the projected cost 
to a single school for a face-to-face setting of the course would be approximately $438 per 



4 

These figures exclude costs, if any, that schools might incur for ( 1 ) ensuring presence of adequate and reliable 
technology to use the program effectively, (2) participant/Literacy Leader stipends and/or costs realized or implied 
by the time teachers divert from other responsibilities to professional development, and (3) Literacy Leader travel 
for the initial workshop/training meeting. Cost categories (2) and (3) presumably would be incurred also in face-to- 
face delivery. Basic training in the use of the technology was provided by Edvantia. 
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person. In both cases, there are the costs that the system would incur if they suspended online 
delivery and switched to a face-to-face delivery mode. In order to compare the full cost of 
delivery of face-to-face instruction with online instruction, development costs must be added to 
these figures to obtain the full projected cost per course seat for face-to-face instruction. Because 
development costs per seat fall as the number of seats increases, an assumption about the level of 
course enrollment must be made to complete the cost analysis. Assuming that course 
development costs for face-to-face delivery are equivalent to online course development 
($59,71 1), and that 896 participants receive instruction in nine groups ($7,009 x 9 = $63,081), 
the full cost per course seat would be projected to be $137. However, such class sizes may be 
unrealistic, owing to traveling distances required to amass willing groups of 100 participants. 
Reducing the assumed class size to 50 participants doubles delivery costs but leaves 
development costs unchanged, for projected costs per course seat of $207. If the face-to-face 
course were offered to individual schools, the projected cost per seat for 896 course seats 
(maintaining the enrollment rate of 16 seats per school) would be $505, or $8,075 per school. 

Summary 

The analysis clearly shows that the online professional development course is 
inexpensive and, when compared to projected costs of face-to-face delivery, cost effective. 
Furthermore, as the number of teachers enrolled in the training increases in coming years, the 
online professional development cost per participant will continue to decrease. 
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Conclusion 



This paper examined the effectiveness of the Tennessee online professional development 
training on three levels: (1) impact on teacher knowledge, (2) impact on student achievement, 
and (3) cost effectiveness of course development and delivery. 

Success of Online Course 

Impact on teacher knowledge. The analyses suggest that the course itself had a positive 
influence on teacher knowledge. Based on a pre- and posttest comparison, teachers’ awareness 
and knowledge clearly increased by a substantial amount (1.6 standard units) following 
participation in the course. 

Impact on student achievement. Understanding the potential impact of professional 
development training on student achievement is a crucial indicator of a professional development 
course’s overall effectiveness. Unfortunately, due to the timing of the release of Tennessee’s 
2005 achievement test scores, the researchers were unable to assess at the time of this report, 
except in a most limited way, the impact of training on student achievement. The analysis of 
2003-2005 AYP data suggests that the reforms implemented in Reading First schools may have 
been associated with improved student outcomes. Due to the timing of gains in relation to the 
online intervention, and given that other efforts to improve student achievement were being 
implemented in these schools, it is unlikely that the online course had an important influence on 
those gains. 

Cost effectiveness of course development and delivery. The analyses suggest that, 
particularly in comparison to projected face-to-face training costs, the Tennessee online 
professional development training is highly cost effective. Furthermore, future training and 
development costs will decrease as the number of participants continues to increase. 

Directions for Future Research 

Due to restricted access to data on instructional practices in reading and the lack of 
availability of public data on state reading tests for 2005, it was not possible to conduct a 
complete analysis of the full impact of the online reading program developed by the Appalachia 
Educational Laboratory for Tennessee. The knowledge base would increase if future research 
could measure the impact of training on classroom practices and student learning outcomes in an 
experimental study. To accomplish this, all stakeholders and participants must agree at the outset 
as to the identification of and subsequent access to data sources. Risks and benefits of the 
research must be known and agreed on by all parties. 

This online professional development project was developed in agreement with the state 
prior to the consideration of completing the cost effectiveness study; therefore, access to 
classroom-level data was not originally sought or granted. Those data were not made available 
for this study. When researchers intend to collect and report data at the classroom level, the 
highly sensitive nature of those data necessitates the involvement of a range of stakeholders to 
approve its use. Increasing the number of stakeholder groups that are involved in data collection 
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and reporting increases the chance for roadblocks to occur. Future studies in school settings 
should heed this lesson and take steps to ensure full access to necessary data prior to 
development and implementation. 
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Appendix 




Table 9 



Research Studies (N= 21) Meeting Selection Criteria for Literati 



Research Study Title Experimental 

Study 

All Learn Mathematics (ALM) 

Assessing One and All (AO A) 

(learning assessment literacy) 

Cognitively Guided Instruction (CGI) * 

(mathematics) 

Collaborative Strategic Reading (CSR) 

Early Literacy and Learning Model (ELLM) 

Eisenhower Professional Development Program 
( longitudinal study of funded professional 
development ) 

Hawaii Algebra Learning Project (HALP) 

Healthy Challenges Project (HCP) 

(tobacco use prevention ) 

High School Algebra Initiative 

Hull Dissertation Study (online discourse analysis) * 

Iowa Chautauqua Program (science) 

Kellogg Dissertation Study (online course support) * 

Mathematics Reform in California 
Mathematics Renaissance 

Peoria Urban Mathematics Plan (PUMP) for Algebra 
Project Creating Independence through Student-owned * 

Strategies (CRISS) (learning strategies) 

Project Law-related Education: Goals for American * 

Leadership (LEGAL) 

Science Education Enhancing the Development of Skills 
(SEEDS) 

Strickland Study on LISTSERVs to Sustain Staff * 

Development 

Student Team Literature (STL) * 

Student Watershed Research Project (SWRP) 

Total of Each Type of Study 7 



e Review 

Quasi- References 

experimental [Primary reference listed first] 

Study 

* University of Illinois at Chicago, 2000; Killion, 2002a 

* Huai, Braden, White, & Elliott, 2003 

Carpenter, Fennema, Peterson, Chaing, & Loef 1989; 
Killion, 2002a 

* Klinger, Vaughn, Arguelles, Hughes, & Leftwich, 2004 

* Wehry, 2001; Killion, 2002a 

* U.S. Department of Education, 2000 



Young, Dougherty, Lai, & Matsumoto, 1998; Killion, 1999 
Hendrix, Gilbert, Kozlowski, Bradley, & Valois, 2002 

Killion, 2002b 
Hull, 2003 

Blunck & Yager, 1996; Killion, 1999, 2002a, 2002b 
Kellogg, 2003 
Cohen & Hill, 1 998 

Acquarelli & Mumme, 1996; Killion, 1999 

Thornton, Langrall, Jones, & Swafford, 2001; Killion, 1999 

Santa, 2004; Killion, 1999, 2002a, 2002b 

Carroll, 1982; Killion, 1999 

Killion, 1999 

Strickland, 2003 

Maclver, Plank, & Balfanz, 1997; Killion, 1999 
Killion, 1999 
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